Hiting anything in air is difficult, you have to gauge distance, height, movement vector and speed of target correctly. For example, if the muzzle speed of your round is 600 m/s and the attacking plane is 700 meters away closing at 350 km/h, you will have to aim at a point some 97 meters ahead of aircraft, adjusted for altitude drop of the round. If you got the speed, height and direction of the plane wrong, then you will miss. You might get some hits based soley on the law of probability, but that takes lot of ammo, the best cure for enemy aircraft were still your own fighters. And as Tiamat mentioned, Japanese 25 mm guns lacked punch, so when Hellcats, well known for their sturdiness, came to strafe the AA batteries the defender might have gotten considerable number of hits, but few kills, because Hellcats could soak up a lot of punishment.
However one important effect the light AA guns, especially when using the tracers, was that the unnerved the hell out of the attacking pilots, reducing their effectivness.
Of course during WWII there were many particularities with the AA defense. Torpedo bombers for example had to do the final part of their attack run low, slow and holding their direction making them easier targets for the defenders and their climb from the attack run made them even better target.
Bombing of the Reich was another one, RAF flying the bomber stream at night and USAAF flying in defens box formations. Gathering info from radar stations and obervation points, the command posts fed directions to AA battery grouping to fill imaginary box in the sky, through which the bombers should pass, with shell - time fused to detonate at that height, firing being done either on time (shells from multiple guns going off at the same time) or continious (all guns firing at their best speed). While bombers flew at preset path, they were instructed to make small direction or height changes at preset intervals, in order to throw off the AA box .
This is where the term flak jacket comes from, USAAF defense box formation made good targets for flak, with shrapnel having no trouble penetrating thin skin of the aircraft, so the aircrew were given heavy protective vests to protect them from flak shrapnel.