Blending Handwriting and Sketching with Computer Code

The world of programming is on the brink of a revolutionary transformation with the introduction of the pen-based interface known as Notate. This innovative tool allows users of computational digital notebooks, such as Jupyter notebooks, to seamlessly integrate hand-drawn diagrams with lines of traditional computer code. By leveraging a sophisticated deep learning model, Notate effectively bridges the gap between handwritten and textual programming contexts, enabling a new level of flexibility and creativity in coding.

One of the key features of Notate is its ability to recognize and interpret handwritten programming symbols, such as variables and functions, and link them to their corresponding typewritten equivalents. This functionality was demonstrated in a case study where users effortlessly sketched quantum circuit diagrams directly within Jupyter notebook code cells. The potential applications of this technology in data science, particularly in sketching plots and charts that interact seamlessly with textual code, are vast and promising.

The groundbreaking work on Notate was recently presented in a paper titled “Notational Programming for Notebook Environments: A Case Study with Quantum Circuits” at the ACM Symposium on User Interface Software and Technology. Lead author Ian Arawjo, a doctoral student in information science, emphasized the need for programming interfaces to evolve and embrace visual elements like images and graphical interfaces to enhance the coding experience.

According to Arawjo, the current programming infrastructure may be limiting the full potential of developers and users alike. By introducing AI-powered, pen-based coding through tools like Notate, a new horizon of possibilities is opening up in the programming landscape. Drawing tablets are increasingly gaining popularity, making the timing of such innovations even more opportune.

Professor Tapan Parikh, an information science expert at Cornell Tech and a co-author of the paper, highlighted the significance of tools like Notate in reshaping our understanding of programming and the role of different representational practices in coding. The collaborative effort behind Notate involved a team of researchers and students, including Anthony DeArmas, Michael Roberts, and Shrutarshi Basu, all contributing to the development and validation of this cutting-edge interface.