Is your country allowed to ban it even if the EU in general allows it?

Allowing something isn't the same as enforcing it to be allowed. If there's regulation, like with ending roaming charges between countries, then it's required to be followed simultaneously across the EU. If there's a directive, like the Working Time Directive, goals of legislation are set out and each member state is required to introduce legislation that implements it. There's also decisions (for one country for one issue), recommendations and opinions (obviously non binding).

There's also the Court of Justice which is the highest court, but only in EU matters. National courts can refer cases to it, or the commission/member states can bring cases against other member states, if they believe they are not following EU law. This would mean either they are not following a regulation, or that the state has not fully/correctly implemented a directive into their own national laws.

As I understand it, there's no specific regulation or directive aimed at gambling itself. There's things tangentially related (data protection, anti money laundering etc). But since there's no regulation or directive saying "gambling must be allowed", there's nothing stopping a member state banning it completely if they so wish.

The only point in which the EU might step in would be if the law was somehow discriminatory or inconsistent (e.g. we ban all foreign gambling sites, but not our own, we ban lottery tickets but not state run casinos, etc).

Germany has regulated it, (though states have slightly different regulations, some states even allowing online gambling, some banning all except the government run lottery).

So it should be possible to regulate it.