You'll need to check the list of the countries that have the visa exemption agreement with Japan. If your country is included in the list, then you will not be required for a visa when entering Japan under normal circumstances. If your country is not included in the list, you'll need to obtain a short-term stay visa from the embassy of Japan in your country.
Please read the Japanese visa guide provided by the ministry of foreign affairs carefully. And be informed that it is the embassy of Japan in your country that makes a decision on issuing a visa. Whether the attendees need Letter of Invitation fully depends on the relations between countries. The conference organizers hereby reiterate that we are neither able to nor authorized to decide the relations between countries.
For more details, please visit: http://www.mofa.go.jp/j_info/visit/visa/index.html.
Important information regarding invitation letters (please read):
In order to qualify for a Letter of Invitation, you must meet both of the following criteria:
Letter of Guarantee is a letter in which an individual or body agrees to be legally and financially responsible for you during your stay in Japan. You may know a willing individual or institution in Japan who is willing to act as a guarantor, or, if you have no connection with the country, the Japanese Ministry of Foreign Affairs may find that it is acceptable to have your university act as the guarantor. Kindly note that the conference organizer cannot provide Letters of Guarantee.