There isn't any absolutely foolproof method to prevent this, since any header information can be spoofed. Session-based tokens are another possible solution, but in that case your javascript is publicly accessible, so anyone who wanted to spend a little time could determine how your token system works and figure out a way around it.
A combination of methods will give you the most wide-ranging protection. You can look for the header, use and .htaccess file, and use tokens. This sort of all-of-the-above approach makes it that much harder to abuse a web server - most abuse comes from people trying to find an easy hole to exploit. The important thing to remember is that you can't become complacent because you've deployed "the best" protection, or because you've got so many layers of protection that it seems impossible to crack. If someone really wanted it bad enough and had the time, they'll find a way. These types of preventative measures are really only deterrents to keep away the lazy, curious, and idly malicious. Targeted attacks are a whole separate class of security, and usually are more centered on server-level security issues.
Sample htaccess. This would not be something you'd put in your root, but rather within a subfolder where you have scripts that should never be called from the address bar:
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www.)?_YOUR_DOMAIN_NAME_HERE.com [NC]
RewriteRule .(php)$ - [NC,F,L]
Check out this article for info about using a token system: https://www.owasp.org/index.php/Cross-Site_Request_Forgery_%28CSRF%29_Prevention_Cheat_Sheet
