Starting with ArcGIS both Python and Visual Basic are currently available to enable customising of the interface. Originally ESRI, ArcGIS developers, had a proprietary programming language which they had developed called Avenue. This was a very popular programming language and was extensively used by many organisations in their implementation of ESRI solutions. However, ESRI decided to switch from Avenue to Microsoft's Visual Basic and a customised version call MapObjects. This customisation option has now been joined by an open source programming language called Python.
MapInfo has always offered MapBasic as the main method of customising MapInfo. Until very recently MapBasic was an optional extra which had to be purchased. Now MapBasic is offered as part of the MapInfo package. MapBasic is, as you would expect, based on the Basic programming language. Within MapInfo it is possible to run MapBasic scripts. Additionally, you can carry out a series of commands in MapInfo and copy the code for use within a separate program.
Quantum GIS can be customised with the Python programming language. Plug ins can be created with Python to perform many diverse GIS applications. Because Quantum GIS is open source it is also possible to even modify the original source code although this would require a knowledge of the C++ programming language.
As previously mentioned all methods of customising GIS systems rely on a knowledge of the relevant programming languages. With MapInfo your current option is limited to learning MapBasic if you wish to customise the software. However, Python is probably the easiest to learn amongst the current programming options available. Considering that both ArcGIS and Quantum GIS offer Python as the method of customisation this would also appear to be the best option to learn if you are new to programming a GIS.