As a rule in France, visiting supporters enter by a different gate for better police management, are segregated in the stadium, and have to wait a bit after the match so that they can't meet the crowd. These preventive measures work well as there are very few incidents, with possibly the exception of matches between Paris and Marseille. However it has a cost as hundreds of mobile security forces are deployed. This state of controlled peace exist also in Spain, Italy. Things are quieter in Germany.
In England, major clubs rose steeply the price of tickets, de facto expelling the working class from stadiums. Now residual violence is localised in second or third UK league matches. There was no risk for you in Chelsea as it is an affluent area of London, but had you been in Millwall wearing a visiting jersey, you might have been in trouble.
There is no more hooliganism in the second or third tiers as there is in the top tier. In fact of the top 10 teams with football related arrests in England 7/10 were for Premiership teams. Some of the slant will be down to attendances though there are some very well attended grounds in the second top tier for sure. The numbers of arrests are very low though...no more than 100 in 2015 for any single club. As a percentage I bet almost every club has a percentage of fan arrests within a similar bracket. Quite simply put, every single club in England, and I bet Europe, has its minute percentage of knob heads.
Chelsea is absolutely a hotbed for football related trouble. A number of fans received banning orders a few months back for racially incited aggravation on an underground tunnel. Chelsea are 6th in the table above. Millwall are 18th (arrest figures for 2015- season).
I've been to Millwall as it happens three times over the past 10-12 years and seen no trouble whatsover. I parked my car in the local streets and walked to and from the game with my scarf on. Trouble at Millwall is NOTHING to do with endemic hooliganism, it's almost all to do with local and historical rivalries.
I have no stats for European grounds but I'd be surprised if France, Germany, the rest of Europe shows any significant lower statistics then in England. Club football is a very safe pastime these days. Almost all arrests are for drunken behaviour rather than hooliganism or violence.
Most violence right now is in Eastern European clubs. And most of the problems over in France right now are caused by Russian thugs, appalling French policing (the French Genderarmerie are absolutely clueless on how to police football fans and have been for many years), and poor English behaviour. And, the ready availability of alcohol. Like a very effective bomb, if you mix all the ingredients correctly it will go off.