FilesExpand file tree

master

/

sigmoid_linear_unit.py

Copy path

Blame

More file actions

Blame

More file actions

Latest commit

 

History

History

History

57 lines (42 loc) · 1.62 KB

master

/

sigmoid_linear_unit.py

Copy path

Top

File metadata and controls

• Code
• Blame

57 lines (42 loc) · 1.62 KB

Raw

Copy raw file

Download raw file

Open symbols panel

Edit and raw actions

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

"""

This script demonstrates the implementation of the Sigmoid Linear Unit (SiLU)

or swish function.

* https://en.wikipedia.org/wiki/Rectifier_(neural_networks)

* https://en.wikipedia.org/wiki/Swish_function

The function takes a vector x of K real numbers as input and returns x * sigmoid(x).

Swish is a smooth, non-monotonic function defined as f(x) = x * sigmoid(x).

Extensive experiments shows that Swish consistently matches or outperforms ReLU

on deep networks applied to a variety of challenging domains such as

image classification and machine translation.

This script is inspired by a corresponding research paper.

* https://arxiv.org/abs/1710.05941

"""

import numpy as np

def sigmoid(vector: np.array) -> np.array:

"""

Mathematical function sigmoid takes a vector x of K real numbers as input and

returns 1/ (1 + e^-x).

https://en.wikipedia.org/wiki/Sigmoid_function

>>> sigmoid(np.array([-1.0, 1.0, 2.0]))

array([0.26894142, 0.73105858, 0.88079708])

"""

return 1 / (1 + np.exp(-vector))

def sigmoid_linear_unit(vector: np.array) -> np.array:

"""

Implements the Sigmoid Linear Unit (SiLU) or swish function

Parameters:

vector (np.array): A numpy array consisting of real

values.

Returns:

swish_vec (np.array): The input numpy array, after applying

swish.

Examples:

>>> sigmoid_linear_unit(np.array([-1.0, 1.0, 2.0]))

array([-0.26894142, 0.73105858, 1.76159416])

>>> sigmoid_linear_unit(np.array([-2]))

array([-0.23840584])

"""

return vector * sigmoid(vector)

if __name__ == "__main__":

import doctest

doctest.testmod()