With regards to stamps: there are some places you can check e.g. some supermarkets or magazine stores like you mentioned, but also a lot of gas stations. Normally they advertise this at the entrance or counter with this logo (I didn’t know if (x)HTML is allowed so here’s the URI):
http://www.post.be/site/fr/_images/residential/customerservice/MagasinTimbre_96x50.gif
The “post(e)” website says there more than 5000 of these shops, so there must be at least a few of them in Brussels. What one cannot understand though is that they don’t have a list of addresses or a map on the post website. Come on it’s 2009? Google maps machups are incredibly easy to write.

They do however have a webshop that delivers the next day (24/7) if you order before 5pm.

@Brusselsblogger about the absence of an amount on belgian stamps: the Post(e) seems to change the stamp format every few months, so nobody really knows how it works. For internal mail (in belgium) there’s 7 different colored (and valued) stamps with numbers from 1 to 7. For intra-EU mail theres 2 different stamps (not 1 and 2, but 1 and 3). There’s a set of 2 stamps similar to the intra-EU ones, but more expensive and with different colors, those are for sending mail outside of the EU.

It’s a really absurd system, because if you have e.g. stamps for sending mail to Belgium and you want to send a letter e.g. to England, you have to go to the mail to buy EU stamps even though you would be able to form the right amount (or a few cents extra) with the Belgian ones. It’s like having one set of currency for buying food, one set for paper supplies and another for buying gas or alcohol.