self_eval#

Functions#

llm_self_eval(input_dir, language, output_dir[, ...])

Module Contents#

self_eval.llm_self_eval(input_dir, language, output_dir, failure_dir=None, llm_name='gpt-4o', evaluation_type='incose', max_prompts=10, overwrite=False, temperature=0.7, collection=None, splitter_type='file', refiner_types=['JanusRefiner'], eval_items_per_request=None, max_tokens=None)#
Parameters:

  • input_dir (typing_extensions.Annotated[pathlib.Path, typer.Option('--input', '-i', help='The directory containing the source code to be evaluated. The files should all be in one flat directory.')]) –

  • language (typing_extensions.Annotated[str, typer.Option('--language', '-l', help='The language of the source code.', click_type=click.Choice(sorted(LANGUAGES)))]) –

  • output_dir (typing_extensions.Annotated[pathlib.Path, typer.Option('--output-dir', '-o', help='The directory to store the evaluations in.')]) –

  • failure_dir (typing_extensions.Annotated[Optional[pathlib.Path], typer.Option('--failure-directory', '-f', help='The directory to store failure files during translation')]) –

  • llm_name (typing_extensions.Annotated[str, typer.Option('--llm', '-L', help="The custom name of the model set with 'janus llm add'.")]) –

  • evaluation_type (typing_extensions.Annotated[str, typer.Option('--evaluation-type', '-e', help='Type of output to evaluate.', click_type=click.Choice(['incose', 'comments']))]) –

  • max_prompts (typing_extensions.Annotated[int, typer.Option('--max-prompts', '-m', help='The maximum number of times to prompt a model on one functional block before exiting the application. This is to prevent wasting too much money.')]) –

  • overwrite (typing_extensions.Annotated[bool, typer.Option('--overwrite/--preserve', help='Whether to overwrite existing files in the output directory')]) –

  • temperature (typing_extensions.Annotated[float, typer.Option('--temperature', '-t', help='Sampling temperature.', min=0, max=2)]) –

  • collection (typing_extensions.Annotated[str, typer.Option('--collection', '-c', help='If set, will put the translated result into a Chroma DB collection with the name provided.')]) –

  • splitter_type (typing_extensions.Annotated[str, typer.Option('-S', '--splitter', help='Name of custom splitter to use', click_type=click.Choice(list(CUSTOM_SPLITTERS.keys())))]) –

  • refiner_types (typing_extensions.Annotated[list[str], typer.Option('-r', '--refiner', help='List of refiner types to use. Add -r for each refiner to use in refinement chain', click_type=click.Choice(list(REFINERS.keys())))]) –

  • eval_items_per_request (typing_extensions.Annotated[int, typer.Option('--eval-items-per-request', '-rc', help='The maximum number of evaluation items per request')]) –

  • max_tokens (typing_extensions.Annotated[int, typer.Option('--max-tokens', '-M', help="The maximum number of tokens the model will take in. If unspecificed, model's default max will be used.")]) –