view runtime/doc/usr_52.txt @ 32936:c517845bd10e v9.0.1776

patch 9.0.1776: No support for stable Python 3 ABI Commit: https://github.com/vim/vim/commit/c13b3d1350b60b94fe87f0761ea31c0e7fb6ebf3 Author: Yee Cheng Chin <ychin.git@gmail.com> Date: Sun Aug 20 21:18:38 2023 +0200 patch 9.0.1776: No support for stable Python 3 ABI Problem: No support for stable Python 3 ABI Solution: Support Python 3 stable ABI Commits: 1) Support Python 3 stable ABI to allow mixed version interoperatbility Vim currently supports embedding Python for use with plugins, and the "dynamic" linking option allows the user to specify a locally installed version of Python by setting `pythonthreedll`. However, one caveat is that the Python 3 libs are not binary compatible across minor versions, and mixing versions can potentially be dangerous (e.g. let's say Vim was linked against the Python 3.10 SDK, but the user sets `pythonthreedll` to a 3.11 lib). Usually, nothing bad happens, but in theory this could lead to crashes, memory corruption, and other unpredictable behaviors. It's also difficult for the user to tell something is wrong because Vim has no way of reporting what Python 3 version Vim was linked with. For Vim installed via a package manager, this usually isn't an issue because all the dependencies would already be figured out. For prebuilt Vim binaries like MacVim (my motivation for working on this), AppImage, and Win32 installer this could potentially be an issue as usually a single binary is distributed. This is more tricky when a new Python version is released, as there's a chicken-and-egg issue with deciding what Python version to build against and hard to keep in sync when a new Python version just drops and we have a mix of users of different Python versions, and a user just blindly upgrading to a new Python could lead to bad interactions with Vim. Python 3 does have a solution for this problem: stable ABI / limited API (see https://docs.python.org/3/c-api/stable.html). The C SDK limits the API to a set of functions that are promised to be stable across versions. This pull request adds an ifdef config that allows us to turn it on when building Vim. Vim binaries built with this option should be safe to freely link with any Python 3 libraies without having the constraint of having to use the same minor version. Note: Python 2 has no such concept and this doesn't change how Python 2 integration works (not that there is going to be a new version of Python 2 that would cause compatibility issues in the future anyway). --- Technical details: ====== The stable ABI can be accessed when we compile with the Python 3 limited API (by defining `Py_LIMITED_API`). The Python 3 code (in `if_python3.c` and `if_py_both.h`) would now handle this and switch to limited API mode. Without it set, Vim will still use the full API as before so this is an opt-in change. The main difference is that `PyType_Object` is now an opaque struct that we can't directly create "static types" out of, and we have to create type objects as "heap types" instead. This is because the struct is not stable and changes from version to version (e.g. 3.8 added a `tp_vectorcall` field to it). I had to change all the types to be allocated on the heap instead with just a pointer to them. Other functions are also simply missing in limited API, or they are introduced too late (e.g. `PyUnicode_AsUTF8AndSize` in 3.10) to it that we need some other ways to do the same thing, so I had to abstract a few things into macros, and sometimes re-implement functions like `PyObject_NEW`. One caveat is that in limited API, `OutputType` (used for replacing `sys.stdout`) no longer inherits from `PyStdPrinter_Type` which I don't think has any real issue other than minor differences in how they convert to a string and missing a couple functions like `mode()` and `fileno()`. Also fixed an existing bug where `tp_basicsize` was set incorrectly for `BufferObject`, `TabListObject, `WinListObject`. Technically, there could be a small performance drop, there is a little more indirection with accessing type objects, and some APIs like `PyUnicode_AsUTF8AndSize` are missing, but in practice I didn't see any difference, and any well-written Python plugin should try to avoid excessing callbacks to the `vim` module in Python anyway. I only tested limited API mode down to Python 3.7, which seemes to compile and work fine. I haven't tried earlier Python versions. 2) Fix PyIter_Check on older Python vers / type##Ptr unused warning For PyIter_Check, older versions exposed them as either macros (used in full API), or a function (for use in limited API). A previous change exposed PyIter_Check to the dynamic build because Python just moved it to function-only in 3.10 anyway. Because of that, just make sure we always grab the function in dynamic builds in earlier versions since that's what Python eventually did anyway. 3) Move Py_LIMITED_API define to configure script Can now use --with-python-stable-abi flag to customize what stable ABI version to target. Can also use an env var to do so as well. 4) Show +python/dyn-stable in :version, and allow has() feature query Not sure if the "/dyn-stable" suffix would break things, or whether we should do it another way. Or just don't show it in version and rely on has() feature checking. 5) Documentation first draft. Still need to implement v:python3_version 6) Fix PyIter_Check build breaks when compiling against Python 3.8 7) Add CI coverage stable ABI on Linux/Windows / make configurable on Windows This adds configurable options for Windows make files (both MinGW and MSVC). CI will also now exercise both traditional full API and stable ABI for Linux and Windows in the matrix for coverage. Also added a "dynamic" option to Linux matrix as a drive-by change to make other scripting languages like Ruby / Perl testable under both static and dynamic builds. 8) Fix inaccuracy in Windows docs Python's own docs are confusing but you don't actually want to use `python3.dll` for the dynamic linkage. 9) Add generated autoconf file 10) Add v:python3_version support This variable indicates the version of Python3 that Vim was built against (PY_VERSION_HEX), and will be useful to check whether the Python library you are loading in dynamically actually fits it. When built with stable ABI, it will be the limited ABI version instead (`Py_LIMITED_API`), which indicates the minimum version of Python 3 the user should have, rather than the exact match. When stable ABI is used, we won't be exposing PY_VERSION_HEX in this var because it just doesn't seem necessary to do so (the whole point of stable ABI is the promise that it will work across versions), and I don't want to confuse the user with too many variables. Also, cleaned up some documentation, and added help tags. 11) Fix Python 3.7 compat issues Fix a couple issues when using limited API < 3.8 - Crash on exit: In Python 3.7, if a heap-allocated type is destroyed before all instances are, it would cause a crash later. This happens when we destroyed `OptionsType` before calling `Py_Finalize` when using the limited API. To make it worse, later versions changed the semantics and now each instance has a strong reference to its own type and the recommendation has changed to have each instance de-ref its own type and have its type in GC traversal. To avoid dealing with these cross-version variations, we just don't free the heap type. They are static types in non-limited-API anyway and are designed to last through the entirety of the app, and we also don't restart the Python runtime and therefore do not need it to have absolutely 0 leaks. See: - https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-c-api - https://docs.python.org/3/whatsnew/3.9.html#changes-in-the-c-api - PyIter_Check: This function is not provided in limited APIs older than 3.8. Previously I was trying to mock it out using manual PyType_GetSlot() but it was brittle and also does not actually work properly for static types (it will generate a Python error). Just return false. It does mean using limited API < 3.8 is not recommended as you lose the functionality to handle iterators, but from playing with plugins I couldn't find it to be an issue. - Fix loading of PyIter_Check so it will be done when limited API < 3.8. Otherwise loading a 3.7 Python lib will fail even if limited API was specified to use it. 12) Make sure to only load `PyUnicode_AsUTF8AndSize` in needed in limited API We don't use this function unless limited API >= 3.10, but we were loading it regardless. Usually it's ok in Unix-like systems where Python just has a single lib that we load from, but in Windows where there is a separate python3.dll this would not work as the symbol would not have been exposed in this more limited DLL file. This makes it much clearer under what condition is this function needed. closes: #12032 Signed-off-by: Christian Brabandt <cb@256bit.org> Co-authored-by: Yee Cheng Chin <ychin.git@gmail.com>
author Christian Brabandt <cb@256bit.org>
date Sun, 20 Aug 2023 21:30:04 +0200
parents 3295247d97a5
children 4635e43f2c6f
line wrap: on
line source

*usr_52.txt*	For Vim version 9.0.  Last change: 2022 Jun 04

		     VIM USER MANUAL - by Bram Moolenaar

		       Write larger plugins

When plugins do more than simple things, they tend to grow big.  This file
explains how to make sure they still load fast and how to split them up in
smaller parts.

|52.1|	Export and import
|52.2|	Autoloading
|52.3|	Autoloading without import/export
|52.4|	Other mechanisms to use
|52.5|	Using a Vim9 script from legacy script

     Next chapter: |usr_90.txt|  Installing Vim
 Previous chapter: |usr_51.txt|  Create a plugin
Table of contents: |usr_toc.txt|

==============================================================================
*52.1*	Export and import

Vim9 script was designed to make it easier to write large Vim scripts.  It
looks more like other script languages, especially Typescript.  Also,
functions are compiled into instructions that can be executed quickly.  This
makes Vim9 script a lot faster, up to a 100 times.

The basic idea is that a script file has items that are private, only used
inside the script file, and items that are exported, which can be used by
scripts that import them.  That makes very clear what is defined where.

Let's start with an example, a script that exports one function and has one
private function: >

	vim9script

	export def GetMessage(count: string): string
	   var nr = str2nr(count)
	   var result = $'To {nr} we say '
	   result ..= GetReply(nr)
	   return result
	enddef

	def GetReply(nr: number): string
	  if nr == 42
	     return 'yes'
	  elseif nr = 22
	     return 'maybe'
	  else
	     return 'no'
	  endif
	enddef

The `vim9script` command is required, `export` only works in a |Vim9| script.

The `export def GetMessage(...` line starts with `export`, meaning that this
function can be called by other scripts.  The line `def GetReply(...` does not
start with `export`, this is a script-local function, it can only be used
inside this script file.

Now about the script where this is imported.  In this example we use this
layout, which works well for a plugin below the "pack" directory:
	.../plugin/theplugin.vim
	.../lib/getmessage.vim

Assuming the "..." directory has been added to 'runtimepath', Vim will look
for plugins in the "plugin" directory and source "theplugin.vim".  Vim does
not recognize the "lib" directory, you can put any scripts there.

The above script that exports GetMessage() goes in lib/getmessage.vim.  The
GetMessage() function is used in plugin/theplugin.vim: >

	vim9script

	import "../lib/getmessage.vim"
	command -nargs=1 ShowMessage echomsg getmessage.GetMessage(<f-args>)

The `import` command uses a relative path, it starts with "../", which means
to go one directory up.  For other kinds of paths see the `:import` command.

How we can try out the command that the plugin provides: >
	ShowMessage 1
<	To 1 we say no ~
>
	ShowMessage 22
<	To 22 we say maybe ~

Notice that the function GetMessage() is prefixed with the imported script
name "getmessage".  That way, for every imported function used, you know what
script it was imported from.  If you import several scripts each of them could
define a GetMessage() function: >

	vim9script

	import "../lib/getmessage.vim"
	import "../lib/getother.vim"
	command -nargs=1 ShowMessage echomsg getmessage.GetMessage(<f-args>)
	command -nargs=1 ShowOther echomsg getother.GetMessage(<f-args>)

If the imported script name is long or you use it in many places, you can
shorten it by adding an "as" argument: >
	import "../lib/getmessage.vim" as msg
	command -nargs=1 ShowMessage echomsg msg.GetMessage(<f-args>)


RELOADING

One thing to keep in mind: the imported "lib/getmessage.vim" script will be
sourced only once.  When it is imported a second time sourcing it will be
skipped, since the items in it have already been created.  It does not matter
if this import command is in another script, or in the same script that is
sourced again.

This is efficient when using a plugin, but when still developing a plugin it
means that changing "lib/getmessage.vim" after it has been imported will have
no effect.  You need to quit Vim and start it again. (Rationale: the items
defined in the script could be used in a compiled function, sourcing the
script again may break those functions).


USING GLOBALS

Sometimes you will want to use global variables or functions, so that they can
be used anywhere.  A good example is a global variable that passes a
preference to a plugin.  To avoid other scripts using the same name, use a
prefix that is very unlikely to be used elsewhere.  For example, if you have a
"mytags" plugin, you could use: >

	g:mytags_location = '$HOME/project'
	g:mytags_style = 'fast'

==============================================================================
*52.2*	Autoloading

After splitting your large script into pieces, all the lines will still be
loaded and executed the moment the script is used.  Every `import` loads the
imported script to find the items defined there.  Although that is good for
finding errors early, it also takes time.  Which is wasted if the
functionality is not often used.

Instead of having `import` load the script immediately, it can be postponed
until needed.  Using the example above, only one change needs to be made in
the plugin/theplugin.vim script: >
	import autoload "../lib/getmessage.vim"

Nothing in the rest of the script needs to change.  However, the types will
not be checked.  Not even the existence of the GetMessage() function is
checked until it is used.  You will have to decide what is more important for
your script: fast startup or getting errors early.  You can also add the
"autoload" argument later, after you have checked everything works.


AUTOLOAD DIRECTORY

Another form is to use autoload with a script name that is not an absolute or
relative path: >
	import autload "monthlib.vim"

This will search for the script "monthlib.vim" in the autoload directories of
'runtimepath'.  With Unix one of the directories often is "~/.vim/autoload".
It will also search under 'packpath', under "start".

The main advantage of this is that this script can be easily shared with other
scripts.  You do need to make sure that the script name is unique, since Vim
will search all the "autoload" directories in 'runtimepath', and if you are
using several plugins with a plugin manager, it may add a directory to
'runtimepath', each of which might have an "autoload" directory.

Without autoload: >
	import "monthlib.vim"

Vim will search for the script "monthlib.vim" in the import directories of
'runtimepath'.  Note that in this case adding or removing "autoload" changes
where the script is found.  With a relative or absolute path the location does
not change.

==============================================================================
*52.3*	Autoloading without import/export

						*write-library-script*
A mechanism from before import/export is still useful and some users may find
it a bit simpler.  The idea is that you call a function with a special name.
That function is then in an autoload script.  We will call that one script a
library script.

The autoload mechanism is based on a function name that has "#" characters: >

	mylib#myfunction(arg)

Vim will recognize the function name by the embedded "#" character and when
it is not defined yet search for the script "autoload/mylib.vim" in
'runtimepath'.  That script must define the "mylib#myfunction()" function.
Obviously the name "mylib" is the part before the "#" and is used as the name
of the script, adding ".vim".

You can put many other functions in the mylib.vim script, you are free to
organize your functions in library scripts.  But you must use function names
where the part before the '#' matches the script name.  Otherwise Vim would
not know what script to load.  This is where it differs from the import/export
mechanism.

If you get really enthusiastic and write lots of library scripts, you may
want to use subdirectories.  Example: >

	netlib#ftp#read('somefile')

Here the script name is taken from the function name up to the last "#". The
"#" in the middle are replaced by a slash, the last one by ".vim".  Thus you
get "netlib/ftp.vim".  For Unix the library script used for this could be:

	~/.vim/autoload/netlib/ftp.vim

Where the function is defined like this: >

	def netlib#ftp#read(fname: string)
		#  Read the file fname through ftp
	enddef

Notice that the name the function is defined with is exactly the same as the
name used for calling the function.  And the part before the last '#'
exactly matches the subdirectory and script name.

You can use the same mechanism for variables: >

	var weekdays = dutch#weekdays

This will load the script "autoload/dutch.vim", which should contain something
like: >

	var dutch#weekdays = ['zondag', 'maandag', 'dinsdag', 'woensdag',
		\ 'donderdag', 'vrijdag', 'zaterdag']

Further reading: |autoload|.

==============================================================================
*52.4*	Other mechanisms to use

Some may find the use of several files a hassle and prefer to keep everything
together in one script.  To avoid this resulting in slow startup there is a
mechanism that only defines a small part and postpones the rest to when it is
actually used.  *write-plugin-quickload*

The basic idea is that the plugin is loaded twice.  The first time user
commands and mappings are defined that offer the functionality.  The second
time the functions that implement the functionality are defined.

It may sound surprising that quickload means loading a script twice.  What we
mean is that it loads quickly the first time, postponing the bulk of the
script to the second time, which only happens when you actually use it.  When
you always use the functionality it actually gets slower!

This uses a FuncUndefined autocommand.  This works differently from the
|autoload| functionality explained above.

The following example shows how it's done: >

	" Vim global plugin for demonstrating quick loading
	" Last Change:	2005 Feb 25
	" Maintainer:	Bram Moolenaar <Bram@vim.org>
	" License:	This file is placed in the public domain.

	if !exists("s:did_load")
		command -nargs=* BNRead  call BufNetRead(<f-args>)
		map <F19> :call BufNetWrite('something')<CR>

		let s:did_load = 1
		exe 'au FuncUndefined BufNet* source ' .. expand('<sfile>')
		finish
	endif

	function BufNetRead(...)
		echo 'BufNetRead(' .. string(a:000) .. ')'
		" read functionality here
	endfunction

	function BufNetWrite(...)
		echo 'BufNetWrite(' .. string(a:000) .. ')'
		" write functionality here
	endfunction

When the script is first loaded "s:did_load" is not set.  The commands between
the "if" and "endif" will be executed.  This ends in a |:finish| command, thus
the rest of the script is not executed.

The second time the script is loaded "s:did_load" exists and the commands
after the "endif" are executed.  This defines the (possible long)
BufNetRead() and BufNetWrite() functions.

If you drop this script in your plugin directory Vim will execute it on
startup.  This is the sequence of events that happens:

1. The "BNRead" command is defined and the <F19> key is mapped when the script
   is sourced at startup.  A |FuncUndefined| autocommand is defined.  The
   ":finish" command causes the script to terminate early.

2. The user types the BNRead command or presses the <F19> key.  The
   BufNetRead() or BufNetWrite() function will be called.

3. Vim can't find the function and triggers the |FuncUndefined| autocommand
   event.  Since the pattern "BufNet*" matches the invoked function, the
   command "source fname" will be executed.  "fname" will be equal to the name
   of the script, no matter where it is located, because it comes from
   expanding "<sfile>" (see |expand()|).

4. The script is sourced again, the "s:did_load" variable exists and the
   functions are defined.

Notice that the functions that are loaded afterwards match the pattern in the
|FuncUndefined| autocommand.  You must make sure that no other plugin defines
functions that match this pattern.

==============================================================================
*52.5*	Using a Vim9 script from legacy script		*source-vim9-script*

In some cases you have a legacy Vim script where you want to use items from a
Vim9 script.  For example in your .vimrc you want to initialize a plugin.  The
best way to do this is to use `:import`.  For example: >

	import 'myNicePlugin.vim'
	call myNicePlugin.NiceInit('today')

This finds the exported function "NiceInit" in the Vim9 script file and makes
it available as script-local item "myNicePlugin.NiceInit". `:import` always
uses the script namespace, even when "s:" is not given.  If "myNicePlugin.vim"
was already sourced it is not sourced again.

Besides avoiding putting any items in the global namespace (where name clashes
can cause unexpected errors), this also means the script is sourced only once,
no matter how many times items from it are imported.

In some cases, e.g. for testing, you may just want to source the Vim9 script.
That is OK, but then only global items will be available.  The Vim9 script
will have to make sure to use a unique name for these global items. Example: >
	source ~/.vim/extra/myNicePlugin.vim
	call g:NicePluginTest()

==============================================================================

Next chapter: |usr_90.txt|  Installing Vim


Copyright: see |manual-copyright|  vim:tw=78:ts=8:noet:ft=help:norl: