First, are you putting this "bottomless pit" on carpet or a hardwood floor? I think a hardwood floor would be easier, because you're starting out with a smoother surface. Flooring also makes a difference as to what materials you can use for the "broken edge" of the pit.
The element that will make or break a bottomless pit is the perspective element of the pit itself. You really want to make sure you're using something that shows the dark depth of the pit.
Something like this (if you're doing a shaft) is a good example.
If you have a good perspective image as your pit, building a slight burm of broken materials around the pit is all you need. When painting a perspective pit, use flat paints for your background colors, especially the deep hole. Some satin or eggshell gloss can be used for highlights near the top of the pit.
If I were building a "blasted throught the floor" pit on my hardwood floor, I would get a section of MDF or particle board, and take time to paint the pit. I would paint several layers to give a good sense of depth. Then I would take some scraps of wood flooring, bust them up, and have some ends sticking up from the floor (mounted to the particle board). I would put enough material around the board to mask the height difference, and I would also paint in a few pieces of flooring that appear to be bent down, dangling into the pit.
Here's a quick little sketch: