I think it's important to first think about what revision you like. osrs, rs3, rs2? If you've made that decision you can go to the download section and select a base you would like to work on, a small note every project will have its downfalls. Nothing is bug proof or complete. They al need work to become something special.
After that you can download Intelij and start the server (most of the items it's by running the launcher.java file). And the client (Loader.java file). When you've done that I would suggest exploring the server you picked, look a bit around, find things you can improve, maybe make a trello board for tracking your content.
Feel free to ask me specific questions if you need more info.