I've got a few gb's with fine gravel. The worms have kept these running for almost 2 years now. I've had a dig and there are not a lot of solids but there are a lot of worms. These gb's have 40mm standpipes so the water flow is strong. I think the defination of aquaponics is still open as we are learning more all the time and there are a lot of different ways to utilize elements of AP. Value adding to the rest of the garden etc.
Plants take a lot of the solids out as well. You only have to look at pot plants and the amount of soil that gets used over time. But as JCO points out there will be a time when the gb's will need to be cleaned. I'm happy to clean them out if they get too bad. I've even worked out a quicker way of doing it. Flood the gb and use a blunt fork to move the media around and something like a post hole digger to move the media around in the deep gb's. The water gets diverted to a filter to catch the solids or just flood the dirt garden with it.